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APPARATUS AND METHOD FOR IMPROVING VISIBILITY IN A MOTOR VEHICLE 
Field Of The Invention 

The invention relates to an apparatus and a method for improving visibility in a motor vehicle 
having at least one infrared-sensitive image sensor system. 

5 Background Information 

The visibility conditions experienced by the driver of a motor vehicle influence the frequency 
of accidents in street traffic, and thus influence traffic safety. In addition to lighting 
conditions, weather conditions determine the driver's ability to see. For example, it is known 
that a higher percentage of fatal traffic accidents happen at night. Fog, rain, and snow also 
10 make visibility conditions worse. Especially poor conditions are present when the lighting 

conditions and the weather conditions are both poor, e.g. when rain reduces visibility at night. 

From German Patent No. 40 32 927, an apparatus is known for improving visibility 
conditions in a motor vehicle having an infrared-sensitive camera and a display device 
1 5 formed as a head-up display. It is proposed that, as information for the driver, the camera 

image be superposed as a virtual image of the outer landscape. In order to illuminate the field 
of view acquired by the infrared-sensitive camera, a radiation source having an infrared 
radiation portion is used. The evaluation of the display is left to the driver. 

20 Summary Of The Invention 

The apparatus and the method of the present invention result in an improvement of the 
visibility in a motor vehicle while at the same time reducing stress on the driver. This 
advantage is achieved by controlling the at least one signaling means for producing the driver 
information, as a fimction of the course of the roadway. Drivers of today's modem motor 

25 vehicles have to process a large amount of information from street traffic and from the motor 
vehicle itself. The apparatus described below and the method enable an advantageous 
reduction in the amount of information that has to be processed by the driver. This results, 
particularly advantageously, in a high degree of acceptance by the driver. In addition, the 
apparatus and the method for reducing the number of accidents in street traffic in poor 
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lighting and/or weather conditions can contribute in particular to a reduction in the number of 
accidents that take place at night. 

Due to the use of spatially high-resolution image sensor systems, for example having a 
5 resolution of 1280x1024 pixels, the number of objects that can be acquired in the surrounding 
environment of the motor vehicle is high. The allocation of the acquired objects in the field of 
detection of the image sensor system to the course of the roadway provides a simple decision 
criterion for reducing the number of objects to be represented. By controlling the at least one 
signaling means for producing the driver information as a function of the position of at least 
10 one object in relation to the course of the roadway, a stimulus overload of the driver is 

advantageously prevented. It is especially advantageous that the apparatus and the method 
described below reduce the distraction of the driver to a minimum. This advantageously 
increases overall traffic safety. In addition, the allocation of the acquired objects to the course 
of the roadway as a decision criterion enables the use of simple, economical processing units. 

15 

The controlling of the signaling means for producing the driver infomiation as a fimction of 
the driving situation and/or the visibility conditions results in additional advantages of the 
apparatus described below and of the method. In this way, a further reduction of the quantity 
of information that a driver of a motor vehicle must process is advantageously achieved by 

20 displaying only information that is important for the driver. For example, the apparatus 

described below and the method enable a situation-specific warning of the driver concerning 
obstacles in the course of the roadway, i.e., in the driver's own lane and in adjacent areas, that 
the driver does not see when traveling at night with low beams. The controlling of the 
signaling means for producing the driver information as a function of the visibility conditions 

25 makes it possible to warn the driver so that he can react in timely fashion to an obstacle. In 
this way, accidents are advantageously prevented. However, at the same time it is possible to 
omit a warning of an obstacle if the obstacle, not visible to the driver, does not present a 
danger. 

30 Items of driver infomiation, suitable for the representation of at least one object and/or of the 
course of the roadway, make it possible for the driver advantageously to register the course of 
the roadway and to rapidly and reliably himself recognize dangers presented by objects. For 
example, at night it is often difficult for the driver to see the course of the roadway itself. 
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Through the simuhaneous representation of the course of the roadway and of objects, a driver 
can judge whether objects represent a danger. 

Through the use of at least one hght pulse in the field of view of the driver in order to warn 
5 him, and/or at least one warning symbol, and/or at least one image marking, and/or at least 
one segment of an image, and/or at least one acoustic signal, and/or at least one optical signal 
as items of driver information, the quantity of information affecting the driver is reduced. 
Through this well-directed warning of the driver via one or more possible information paths, 
the danger of distracting the driver is advantageously reduced. 

10 

By using at least one source of infirared radiation to illuminate at least a part of the 
surroimding environment of the motor vehicle, acquired by the infirared-sensiti ve image 
sensor system, the sensitivity and the spatial area of acquisition of the image sensor system 
are advantageously increased. The infi-ared radiation source makes it possible for the at least 
15 one image sensor system to acqmre objects that do not give off infi-ared radiation. 

A computer program having program code means is especially advantageous for the 
execution of all steps of the method described below, if the program is executed on a 
computer. The use of a computer program enables the rapid and economical adaptation of the 
20 method to different image sensor systems and/or signaling means. 

Brief Description Of The Drawings 

Figure 1 shows an overview drawing of a first exemplary embodiment for improving the 
visibility in a motor vehicle. 

25 

Figure 2 shows an overview drawing of a second exemplary embodiment for improving the 
visibility in a motor vehicle. 

Figure 3 shows a block diagram of the apparatus for improving the visibility in a motor 
30 vehicle. 

Figure 4 shows a flow diagram. 
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Detailed Description 

In the following, an apparatus and a method for improving the visibility in a motor vehicle 
having an infrared-sensitive image sensor system are described. Via a signaling means, for 
example a monitor or a head-up display, the driver of the motor vehicle is given driver 
5 information concerning the course of the roadway and/or objects in the surrounding 

environment of the motor vehicle. The displaying of the driver information here takes place 
as a function of the course of the roadway. 

Figure 1 shows an overview drawing of a first exemplary embodiment for improving the 

10 visibility in a motor vehicle 10, made up of an infrared-sensitive image sensor system 12, a 
headlight 16, and an image on a display screen 26. Image sensor system 12 is situated in the 
interior of motor vehicle 10, behind the windshield, in the area of the interior rearview mirror. 
Image sensor system 12 is oriented in such a way that area of acquisition 14 of image sensor 
system 12 extends to the surrounding environment of motor vehicle 10 in the direction of 

1 5 travel. Image sensor system 1 2 includes an infrared-sensitive video camera having a CMOS 
image sensor and/or a CCD image sensor. The image sensor of image sensor system 12 
acquires at least near infrared radiation in the wavelength range between 780 nm and 1000 
nm. Motor vehicle 1 0 is driven by a driver, motor vehicle 1 0 being situated on a street and 
traveling in the direction of travel. In the situation shown in Figure 1, the weather conditions 

20 and the lighting conditions are poor, because darkness is adversely affecting the driver's view. 
Two headlights 16, situated at the right and at the left in the front area of motor vehicle 10, in 
the vicinity of the bimiper, illuminate the surroimding environment of motor vehicle 10 in the 
direction of travel. In Figure 1, in a simplified representation only one headlight 16 is shown. 
In addition to the low-beam light 18 in the visible light spectral range, headlights 16 also 

25 produce radiation having a high-beam characteristic 20 in the infrared spectral range. The 

range of low-beam 18 is approximately 40 meters. Headlights 16 have a high-beam fimction 
in the visible spectral range, with which the driver can see up to 200 meters, depending on 
weather conditions. In this exemplary embodiment, the high-beam fimction in the visible 
spectral range is not activated. The radiated spectnun of the halogen lamps of headlights 16 

30 contain a large infrared portion that is radiated by these modified headlights 16 with a high- 
beam characteristic, in a manner invisible to human beings. The drivers perception 22 is 
strongly limited by the darkness. In contrast, infrared-sensitive image sensor system 12 
achieves a better perception 24 through the modification of front headlights 1 6, having 
radiation at least of near infrared radiation with wavelengths between 780 and 1000 nm with 
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high-beam characteristic 20. In the perception 24 of infrared-sensitive image sensor system 
12, an oncoming motor vehicle 28, a pedestrian 30, and roadway 32 can be seen, while in 
driver's perception 22 only headlights 29 of oncoming motor vehicle 28 can be recognized. In 
perception 24 of image sensor system 12, pedestrian 30, crossing behind oncoming motor 
5 vehicle 28, can be recognized clearly, while in the driver's perception 22 he is not visible. 
Perception 24 of infrared-sensitive image sensor system 12 is processed and is displayed, as 
an image on a display screen 26, to the driver of motor vehicle 1 0. As a display screen, a 
monitor is used that is situated in the dashboard of motor vehicle 10. This display screen can 
be situated in the area of the center console and/or in the multi-instrument panel of motor 

10 vehicle 10, immediately behind the steering wheel. Besides the displaying of the speed and/or 
RPM by the multi-instrument panel, it is also possible for the driver to see the image of the 
surrounding environment of motor vehicle 10 without changing the direction of his gaze. 
With the aid of suitable image processing algorithms, objects 28, 30 in the field of detection, 
i.e. in the area of the course of the roadway, are recognized, and objects 28, 30 are allocated 

15 to the course of the roadway. Roadway 32 here fundamentally includes the driver's lane and 
the lane of the oncoming traffic. On highways, roadway 32 is formed by at least the driver's 
lanes. Roadway 32 is defined by roadway markings such as guideposts and/or lane marking 
lines. The course of the roadway here includes roadway 32 itself and areas adjacent to 
roadway 32, such as for example edge strips and/or footpaths and/or bicycle paths and/or 

20 entrances of streets. In this exemplary embodiment, an oncoming motor vehicle 28 and a 
pedestrian 30 are recognized as objects 28, 30. In the image on display screen 26, the image 
recorded by infrared-sensitive image sensor system 12 of the surrounding environment of 
motor vehicle 10 is displayed with a marking 34 of oncoming motor vehicle 28, a marking 36 
of pedestrian 30, and a marking of the course 38 of roadway 32. This image on display screen 

25 26, with inserted markings 34, 36, 38, is shown to the driver of motor vehicle 10. Through 
markings 34, 36, 38, the driver recognizes objects 28, 30 significantly faster than would be 
the case if only the unprocessed image information of infrared-sensitive image sensor system 
12 were displayed, and can easily allocate them to his driving lane. In this exemplary 
embodiment, the image processing algorithms are designed so that the course of the roadway 

30 38 is always marked, while markings 34, 36 of objects 28, 30 are executed only if objects 28, 
30 are situated in the area of the course of the roadway of motor vehicle 10. In an 
embodiment, the optical warning via markings 34, 36, 38 is additionally supported by an 
acoustic waming if a dangerous situation is recognized. Via a loudspeaker, the driver of 
motor vehicle 1 0 can be additionally wamed acoustically. In this exemplary embodiment, a 
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dangerous situation is recognized because pedestrian 30 is crossing roadway 32 behind 
oncoming motor vehicle 28, and there is the danger of a colhsion of the driver's vehicle 10 
with pedestrian 30. 

5 Figure 2 shows an overview drawing of a second exemplary embodiment for improving 
visibility in a motor vehicle 10, made up of an infrared-sensitive image sensor system 12, a 
headlight 16, and the actual view of driver 40 of motor vehicle 10. As in the first exemplary 
embodiment according to Figure 1 , image sensor system 1 2 is situated in the interior of motor 
vehicle 1 0, behind the windshield, in the area of the interior rearview mirror. Image sensor 

10 system 12 is oriented in such a way that area of acquisition 14 of image sensor system 12 

extends to the surroimding environment of motor vehicle 10 in the direction of travel. Image 
sensor system 12 includes an infirared-sensitive video camera having a CMOS image sensor 
and/or a CCD image sensor. The image sensor of image sensor system 12 acquires at least 
near infirared radiation in the wavelength range between 780 nm and 1000 nm. Motor vehicle 

15 10 is driven by a driver, motor vehicle 10 being situated on a street and traveling in the 

direction of travel. The same driving situation is present as in the first exemplary embodiment 
according to Figure 1 . The weather conditions and the lighting conditions are poor, because 
darkness is adversely affecting the driver's view. Two headlights 1 6, situated at the right and 
at the left in the firont area of motor vehicle 10, in the vicinity of the bumper, illuminate the 

20 surroimding environment of motor vehicle 10 in the direction of travel. In Figure 2, in a 

simplified representation only one headlight 1 6 is shown. In addition to the low-beam light 
18 in the visible spectral range, headlights 16 also produce radiation having a high-beam 
characteristic 20 in the infirared spectral range. The range of low-beam light 18 is 
approximately 40 meters. Headlights 16 have a high-beam fiinction in the visible spectral 

25 range, with which the driver can see up to 200 meters, depending on weather conditions. In 
this exemplary embodiment, the high-beam fimction in the visible spectral range is not 
activated. The radiated spectrum of the halogen lamps of headlights 16 contain a large 
infi-ared portion that is radiated by these modified headlights 16 with a high-beam 
characteristic, in a manner invisible to human beings. The driver's perception 22 is strongly 

30 limited by the darkness. In contrast, infrared-sensitive image sensor system 12 achieves a 

better perception 24 through the modification of front headlights 16, having radiation at least 
of near infrared radiation with wavelengths between 780 and 1000 nm with high-beam 
characteristic 20. In the perception 24 of infrared-sensitive image sensor system 12, an 
oncoming motor vehicle 28, a pedestrian 30, and roadway 32 can be seen, while in driver's 
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perception 22 only headlights 29 of oncoming motor vehicle 28 can be recognized. In 
perception 24 of image sensor system 12, pedestrian 30, crossing behind oncoming motor 
vehicle 28, can be seen clearly, while in driver's perception 22 he is not visible. Figure 2 
additionally shows the actual view of driver 40, including steering wheel 42, windshield 44, 
and dashboard 46. In this second exemplary embodiment, perception 24 of infrared-sensitive 
image sensor system 12 is supplied to a processing unit that produces a warning only if a 
dangerous situation is recognized. With the aid of suitable image processing algorithms, 
objects 28, 30 in the field of detection, i.e. in the area of the roadway, are recognized, and 
objects 28, 30 are allocated to the course of the roadway. As in the exemplary embodiment 
according to Figure I, roadway 32 here fundamentally includes the driver's lane and the lane 
of the oncoming traffic. On highways, roadway 32 is formed by at least the driver's lanes. 
Roadway 32 is defined by roadway markings such as guideposts and/or lane marking lines. 
The course of the roadway here includes roadway 32 itself and areas adjacent to roadway 32, 
such as for example edge strips and/or footpaths and/or bicycle paths and/or entrances of 
streets. In this exemplary embodiment, an oncoming motor vehicle 28 and a pedestrian 30 are 
recognized as objects 28, 30. The processing unit recognizes the dangerousness of the 
situation. Via a simple projection device, in this exemplary embodiment a simple head-up 
display, on windshield 44 a small warning symbol 34, 36 is produced as marking 34 of 
oncoming motor vehicle 28 and as marking 36 of pedestrian 30 in the direction of view of the 
driver, at the position, thus determined, of oncoming motor vehicle 28 and of pedestrian 30. 
In this way, the driver is made to direct his gaze in the direction in which objects 28, 30 will 
later actually appear to the driver as obstacles in his field of view. In this exemplary 
embodiment, a colored marking in the form of a red and/or yellow triangle is used as warning 
symbol 34, 36. Altematively, it is possible, using a short light impulse, to cause the driver to 
direct his g2ize in the direction of the obstacle, using a projection device. The view of driver 
40 of the surrounding environment of motor vehicle 1 0 according to Figure 2 accordingly 
includes the light 48 from the headlights of oncoming motor vehicle 28, a marking 34 of 
oncoming motor vehicle 28, and a marking 36 of pedestrian 30. These markings 34, 36 direct 
the attention of the driver of motor vehicle 10 to these objects 28, 30. The image processing 
algorithms are designed so that the marking 34, 36 of objects 28, 30 takes place only if 
objects 28, 30 are situated in the area of the course of the roadway of motor vehicle 10, and if 
a dangerous situation is present. In this exemplary embodiment, a dangerous situation is 
recognized because pedestrian 30 is crossing roadway 32 behind oncoming motor vehicle 28, 
and there is the danger of a coUision of the driver's vehicle 10 with pedestrian 30. 
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In the two exemplary embodiments according to Figures 1 and 2, image sensors are used that 
have a high resolution. Usable semiconductor image recording chips have resolutions that are 
sufficient for a satisfactory image representation and that enable object recognition at 
5 distances up to approximately 70 meters. A recognition of objects located fiirther than 70 
meters jfrom the image sensor requires higher-resolution image sensors (imagers) having 
standard resolutions with 1024x768 pixels or 1280x1024 pixels. With standard resolutions of 
1024x768 pixels, an object recognition up to approximately 1 10 meters is possible, while 
with a standard resolution of 1280x1024 pixels an object recognition up to approximately 140 

10 meters can be carried out. In the two exemplary embodiments according to Figures 1 and 2, 
the coating of the camera optics is adapted to the spectral range used. The coating is designed 
so that the optical characteristics in the visible spectral range are not significantly worsened. 
In this way, is possible also to use the image sensor system for other fiinctions in daylight, 
i.e., in the visible spectral range. In addition, the aperture of the optics is adapted to the 

1 5 prevailing dark sensitivity. 

Figure 3 shows a block diagram of the first and second exemplary embodiments, 
corresponding to Figures 1 and 2, of the apparatus for improving visibility in a motor vehicle, 
made up of an infrared-sensitive image sensor system 12, a processing unit 62, and at least 

20 one signaling means 66. Infrared-sensitive image sensor system 12 acquires optical signals 
from the surrounding environment of the motor vehicle in the form of image data. Via signal 
line 60, the optical signals are transmitted electrically and/or optically from infrared-sensitive 
image sensor system 12 to processing unit 62. Alternatively, or additionally, transmission by 
radio is possible. Processing unit 62 is made up of module 72, shown in Figure 4, which in 

25 these exemplary embodiments is realized as programs of at least one microprocessor. In this 
exemplary embodiment, processing unit 62 is physically separated from the other 
components 12, 66. Altematively, it is possible for processing imit 62 to form a imit together 
with image sensor system 12, or for processing unit 62 to be housed in signaling means 66. 
Processing xmit 62 calculates, from the optical signals of infrared-sensitive image sensor 

30 system 12, signals for driver information. The calculated signals for driver information are 

electrically and/or optically transmitted to at leeist one signaling means 66 via a signal line 64. 
Altematively, or additionally, a transmission by radio is possible. From the signals for driver 
information, signaling means 66 produce the actual driver information, for example in the 
form of an optical and/or an acoustic waming. As signaling means 66, in the first exemplary 
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embodiment a display in the multi-instrument panel is used as a first signaling means 66, and 
a loudspeaker is used as a second signaling means 66, while in the second exemplary 
embodiment a projection device is used. 

5 Figure 4 shows a flow diagram of the first and second exemplary embodiments, 

corresponding to Figures 1 and 2, of the method for improving visibility in a motor vehicle, 
and of the apparatus according to Figure 3, made up of processing module 72. Optical signals 
70 are supplied to processing module 72, which calculates, as output signals, the signals for 
driver information 74. The processing module is made up of two modules that work in 

10 parallel; these are the module for roadway course recognition 76 and the module for object 
recognition 78. The algorithms for lane and/or roadway recognition of the module for 
roadway course recognition 76 and for object recognition of object recognition module 78 are 
joined to form an overall algorithm. These two modules 76, 78 exchange information and 
partial results during the processing. In the module for roadway course recognition 76, from 

15 optical signals 70 objects are determined that define the roadway or the lane. These objects 
are for example guideposts and/or lane marking lines. The course of the roadway is 
calculated using the knowledge of the position of these objects. The objects for calculating 
the course of the roadway are determined by evaluating contrasts in the image. For example, 
in order to determine the l£tne marking lines, contrasts differences between the lane markings 

20 and the roadway surface are evaluated and are traced by the processing unit in the image 

sequence. Irregularities and/or brief interruptions of the lines are corrected, for example using 
Kalman filtering algorithms. In the module for object recognition 78, objects are likewise 
determined from the optical signals. For the object detection, contrast differences between the 
object and its sxirroundings are evaluated. In a variant of the described method, a 

25 determination of the distance of the detected objects is carried out using a stereo camera. 

From the size of the outline of the object and its distance, it is possible to determine the size 
of the object. Objects that are recognized include in particular oncoming motor vehicles 
and/or pedestrians and/or bicyclists and/or motorcyclists and/or trucks. In a first variant, 
objects are merely recognized as such, while in a further variant an object classification is 

30 carried out. A comparison between a recognized image pattern, for example the shape and/or 
the size of the object, and an image pattern stored in the processing unit forms the basis for 
the object classification. Subsequently, a calculation of the three-dimensional position of the 
recognized objects in relation to the determined course of the roadway is carried out. Using 
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this overall algorithm, it is possible to carry out an allocation of an object to the course of the 
roadway, in particular to the lane and/or to the roadway. 

The described apparatus and the method are not limited to a single image sensor system. 
5 Rather, in a variant additional image sensor systems are used whose optical signals are 

supplied to the at least one processing unit. Here, the image sensor systems are equipped with 
color image sensors and/or black-and-white image sensors. In an additional variant, at least 
one image sensor system is used that is made up of at least two image sensors that record 
essentially the same scene. In another variant, at least one stereo camera is used. 

10 

In a further variant of the described apparatus and of the method, besides at least one like 
pulse and/or at least one warning symbol and/or at least one image marking and/or an 
acoustic signal, altematively or additionally at least one segment of an image and/or a haptic 
signal are produced as driver information. These possibilities are used either individually or 
15 in arbitrary combinations. 

A further variant of the described apparatus contains, besides the at least one optical signaling 
means and/or the at least one acoustic signaling means, altematively or additionally at least 
one haptic signaling means. These signaling means are used either individually or in arbitrary 
20 combinations. 

In a further variant of the described apparatus and of the method, the processing unit 
altematively or additionally takes into account the visibility conditions. The visibility 
conditions are defined here by the view of the driver. In this variant, the inclusion of the 
visibility conditions means that the driver is warned only of objects that he himself cannot 
25 see. This results in a reduction of the quantity of information that has to be processed by the 
driver. 

A further variant provides that, instead of the modified headlight with halogen lamp 
according to one of the preceding exemplary embodiments, an infrared-laser-based headlight 
30 is used as a source of infrared radiation. 

In a further variant of the described apparatus and of the method, more than one processing 
unit is used. In this way, a distribution of the algorithms to a plurality of processing units is 
possible. At the same time, there is a redundancy of the required computing capacity, so that 
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when there is a failure of a processing unit, the apparatus and the method for improving 
visibiUty continue to remain capable of functioning, because the remaining processing units 
compensate for the failure. 

In a further variant of the described apparatus and of the method, the use of at least one 
additional sensor enables an improvement of the visibility in a motor vehicle. Through the 
use of at least one additional sensor and the fusion of the sensor signals with the produced 
optical signals, a more reliable recognition of the course of the roadway and/or of the objects 
is attained. As the at least one additional sensor, at least one radar sensor and/or at least one 
ultrasonic sensor and/or at least one LIDAR distance sensor are used. The use of at least one 
additional sensor enables the redundant determination of the position of at least one object 
and/or of the course of the roadway. 
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